Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Distinguish log file names and limit log file count #113

Closed
wants to merge 13 commits into from

Conversation

acelyc111
Copy link
Owner

No description provided.

@acelyc111 acelyc111 changed the base branch from master to dis_servers May 20, 2024 03:59
@acelyc111 acelyc111 force-pushed the dis_log_dirs branch 2 times, most recently from 0f7caaf to 4b312be Compare May 21, 2024 01:53
@acelyc111 acelyc111 changed the title Dis log dirs feat: Distinguish log file names and limit log file count May 21, 2024
@acelyc111 acelyc111 force-pushed the dis_log_dirs branch 2 times, most recently from ebdf3f5 to e376e1a Compare May 21, 2024 02:00
This patch adds a new flag `--separate_servers` to indicate whether to build
`pegasus_collector`,`pegasus_meta_server` and `pegasus_replica_server` binaries
separately, otherwise a combined `pegasus_server` binary will be built. The
corresponding option in CMake is `SEPARATE_SERVERS`.
acelyc111 added 2 commits May 22, 2024 11:22
There are 2 options stats_dump_period_sec and stats_persist_period_sec in
RocksDB to adjust the period to dump/persist stat, each RocksDB instance
creates 2 dependent threads for this aim.

A replica server is possible to serve thousands of replicas, it means
thousands of RocksDB instances are possible to be running in one process,
then thousands of threads will be created will may consume much resouce.

This patch introduces 2 options, then it's possible to disable the threads
by setting them to 0.
Fix build failures like:

1.
```
/bin/ld: /.../thirdparty/output/lib/libcurl.a(libcurl_la-content_encoding.o): in function `brotli_close_writer':
content_encoding.c:(.text+0x2c): undefined reference to `BrotliDecoderDestroyInstance'
```

2.
```
/bin/ld: /.../thirdparty/output/lib/libcurl.a(libcurl_la-easy.o): relocation R_AARCH64_ADR_PREL_PG_HI21 against symbol `malloc' which may bind externally can not be used when making a shared object; recompile with -fPIC
/bin/ld: /.../thirdparty/output/lib/libcurl.a(libcurl_la-easy.o)(.text+0x30): unresolvable R_AARCH64_ADR_PREL_PG_HI21 relocation against symbol `malloc'
```

3.
```
/bin/ld: idn.c:(.text+0x34): undefined reference to `idn2_lookup_ul'
```
acelyc111 and others added 4 commits May 22, 2024 19:46
…pache#2016)

fix(aarch64): Set the correct LD_LIBRARY_PATH for aarch64 platforms
Bumps google.golang.org/protobuf from 1.29.1 to 1.33.0.
…ion (apache#2019)

Use the current working directory rather than the fixed '/home/work/' directory.
acelyc111 added 2 commits May 24, 2024 15:26
Add a new configuration 'log_dir' to indicate the directory to place the
logs especially. 'data_dir' will be used if it's empty.

```diff
[core]
+log_dir = 
```
Mainly including:
- Using fmt::format/print instead of sprintf/printf, which is more convenient
  and faster
- Add some configuration validators
- Encapsulate print_header(), print_long_header() and print_body() for
  screen_logger and simple_logger
- Add more comments
- Naming improvment
acelyc111 added 3 commits May 27, 2024 11:58
…size and total count (apache#2024)

- A new symlink is added to point to the latest log file, the symlink is in
  the same directory as the log files.

**Behavior change:**
- Log file is rolled by `max_log_file_bytes` configuration (64MB by default)
  instead of the fixed line count, i.e. 200000.
- Log files are named as `log.<yyyyMMdd_hhmmss_SSS>`, e.g. log.20240524_1806_235
  instead of `log.<index>.txt`, e.g. log.123.txt

```diff
[tools.simple_logger]
+max_log_file_bytes =
```
Move lookup_hostname() from utils/utils.h to rpc_host_port, and remove the use
less addr2host() in src/base/pegasus_utils.h

There is no functionality changes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant